DATA ENGINEERING + AI / END-TO-END PLATFORM

The job market, unified.
Matched by meaning, not keywords.

Polaris collects Data & AI job postings from three Vietnamese recruitment sources, transforms raw web pages into a traceable PostgreSQL model, and ranks CV-to-job fit with vector embeddings — served through a Next.js portal with market dashboards and scheduled alerts.

View source code ↗

ROLE

Data + Full-stack Engineer

SCOPE

Individual project · Jun 2026

SOURCES

TopCV · LinkedIn · ITViec

STACK

Python · Dagster · dbt · PostgreSQL + pgvector · Next.js · MinIO · Redis

01 / CONTEXT & PROBLEM

Three sources, three formats, and keyword search that misses the point.

Vietnam's data-job market is scattered across boards with incompatible structures, duplicate postings, and no way to ask the question that matters: "which of these jobs actually fits my CV?" Keyword matching rewards buzzword overlap, not real fit.

The goal: one traceable data model where every normalized record can be traced back to its preserved raw HTML, plus semantic matching that ranks jobs by meaning — wrapped in a portal a job-seeker would actually use.

02 / ARCHITECTURE

Raw HTML to ranked match

3 job boards

daily crawls via GitHub Actions

→

MinIO raw layer

original HTML preserved · full traceability

→

BeautifulSoup parsers

source-specific fields · transactional upserts

→

PostgreSQL + dbt + pgvector

staging & analytical models · 384-dim MiniLM embeddings · cosine similarity

→

Next.js portal

AI matching · dashboards · hourly alerts

Dagster orchestrates · Redis distributed locks prevent duplicate runs · 9-container Docker Compose stack incl. Cube, Cube Store, Metabase

03 / KEY DECISIONS

What I chose, and why

Preserve raw HTML before parsing anything

Job boards change their markup constantly. Keeping the original page in MinIO means a parser bug is a re-run, not data loss — and every normalized field stays auditable back to its source.

pgvector inside PostgreSQL, not a separate vector DB

Embeddings live next to the relational model, so semantic ranking joins directly with structured filters (location, level, salary) in one SQL query — no sync problem between two stores.

Redis distributed locks around scheduled jobs

Daily crawls and hourly alert deliveries are guarded by locks and authenticated internal APIs — overlapping runs can't double-send emails or double-insert postings.

Quality gates across both languages

21 automated tests cover TypeScript and Python: linting, formatting, dbt parsing, Prisma validation, and Dagster definitions — the polyglot stack fails fast in CI, not in production.

04 / RESULTS

Measured outcomes

3 → 1

recruitment sources unified into a single traceable PostgreSQL model

384-dim

semantic ranking with MiniLM embeddings instead of keyword matching

9

containers operated healthy: DB, cache, orchestration, analytics, web

21

automated tests & quality gates across TS, Python, dbt, Prisma, Dagster

A recorded dataset of 234 postings visualized by location, experience level, hiring trend, and company demand — with daily crawling and hourly SMTP-tracked alert delivery running unattended.

05 / WHAT I'D DO DIFFERENTLY

Evaluate the matching, not just build it.

Semantic ranking works, but "works" needs a benchmark: a labeled set of CV-job pairs to measure precision against keyword baselines, and A/B evidence that embedding similarity predicts real fit. That evaluation harness comes before any model upgrade.

I'd also grow the dataset — 234 recorded postings proves the pipeline; market-level claims need months of accumulation, which the daily automation is designed to deliver.